Fix the search, similar, list, and developer methods #557

mmoksh · 2022-06-09T04:50:16Z

Google introduced new mapping for the first page of the search results to highlight one app that matches the search query. The rest of the pages have the same mapping as previous.

In this PR, I fixed gplay.search script by updating mappings. I also included the changes from PR #545 that fixes gplay.app in this PR to avoid git conflicts when merging both PRs.

Edit: Added a few more commits to fix the gplay.similar script, and to handle special cases when there is no search results.

Edit2: Added a new commit to fix the gplay.list method.

Use this for testing:

"google-play-scraper": "git+https://github.com/facundoolano/google-play-scraper#pull/557/head"

Aliaksandr-Kasko-JazzTeam · 2022-06-14T06:40:16Z

Because of changes to GPlay you should update tests to fix failing checks.

Aliaksandr-Kasko-JazzTeam · 2022-06-14T14:16:05Z

There's no fails with list method, but it returns an empty array for all combinations of category and collection :-\

NathanManning · 2022-06-15T17:56:33Z

@mmoksh, did something change? I'm getting empty results again for the clustered apps.

gplay .search({ term: 'facebook', }) .then(console.log, console.log)

returns []

mmoksh · 2022-06-16T01:51:37Z

@NathanManning I just checked! It works just fine!
@Aliaksandr-Kasko-JazzTeam This PR does not contain a fix for the gplay.list method. It only fixes the gplay.search and gplay.similar methods. I'll try to update the unit tests later this week.

Aliaksandr-Kasko-JazzTeam · 2022-06-16T06:38:49Z

@mmoksh could you please fix gplay.list method too? It's really necessary to us

NathanManning · 2022-06-17T16:22:59Z

@NathanManning I just checked! It works just fine! @Aliaksandr-Kasko-JazzTeam This PR does not contain a fix for the gplay.list method. It only fixes the gplay.search and gplay.similar methods. I'll try to update the unit tests later this week.

@mmoksh My bad. There's something funky going on in our package-lock.

mmoksh · 2022-06-18T08:10:46Z

@Aliaksandr-Kasko-JazzTeam The gplay.list method is fixed now. You might need to delete the package-lock.json file and the node_modules folder to make sure your app is not using an outdated version of this PR.

ninjachen · 2022-06-20T03:12:54Z

Hello, I'm trying to use the new list function of this PR and meet an error, could you help

The steps to reproduce

➜  scraper git:(master) ✗ npm -v
7.24.0
➜  scraper git:(master) ✗ rm -rf node_modules 
➜  scraper git:(master) ✗ rm package-lock.json 
➜  scraper git:(master) ✗ npm install
➜  scraper git:(master) ✗ npx functions-framework --target=list

Serving function...
Function: list
Signature type: http
URL: http://localhost:8080/
Error: Invalid collection topselling_free
    at validate (/Users/ninja/src/lab/scraper/node_modules/google-play-scraper/lib/list.js:43:11)
    at /Users/ninja/src/lab/scraper/node_modules/google-play-scraper/lib/list.js:14:5
    at new Promise (<anonymous>)
    at Object.list (/Users/ninja/src/lab/scraper/node_modules/google-play-scraper/lib/list.js:13:10)
    at exports.list (/Users/ninja/src/lab/scraper/index.js:31:15)
    at /Users/ninja/src/lab/scraper/node_modules/@google-cloud/functions-framework/build/src/invoker.js:98:17
    at processTicksAndRejections (node:internal/process/task_queues:76:11)

package.json

➜  scraper git:(master) ✗ cat package.json 
{
  "name": "scraper",
  "version": "0.5.1",
  "dependencies": {
    "google-play-scraper": "git+https://github.com/facundoolano/google-play-scraper#pull/557/head"
  },
  "scripts": {
    "start": "npx functions-framework --target=app --signature-type=http"
  },
  "devDependencies": {
    "@google-cloud/functions-framework": "1.9.0"
  }
}

Aliaksandr-Kasko-JazzTeam · 2022-06-20T12:47:43Z

@mmoksh many thanks to you. It seems to work fine. 👍

@facundoolano This PR looks great to merge into master and new package version

mmoksh · 2022-06-22T02:16:24Z

Hello, I'm trying to use the new list function of this PR and meet an error, could you help

The steps to reproduce

➜  scraper git:(master) ✗ npm -v
7.24.0
➜  scraper git:(master) ✗ rm -rf node_modules 
➜  scraper git:(master) ✗ rm package-lock.json 
➜  scraper git:(master) ✗ npm install
➜  scraper git:(master) ✗ npx functions-framework --target=list

Serving function...
Function: list
Signature type: http
URL: http://localhost:8080/
Error: Invalid collection topselling_free
    at validate (/Users/ninja/src/lab/scraper/node_modules/google-play-scraper/lib/list.js:43:11)
    at /Users/ninja/src/lab/scraper/node_modules/google-play-scraper/lib/list.js:14:5
    at new Promise (<anonymous>)
    at Object.list (/Users/ninja/src/lab/scraper/node_modules/google-play-scraper/lib/list.js:13:10)
    at exports.list (/Users/ninja/src/lab/scraper/index.js:31:15)
    at /Users/ninja/src/lab/scraper/node_modules/@google-cloud/functions-framework/build/src/invoker.js:98:17
    at processTicksAndRejections (node:internal/process/task_queues:76:11)

package.json

➜  scraper git:(master) ✗ cat package.json 
{
  "name": "scraper",
  "version": "0.5.1",
  "dependencies": {
    "google-play-scraper": "git+https://github.com/facundoolano/google-play-scraper#pull/557/head"
  },
  "scripts": {
    "start": "npx functions-framework --target=app --signature-type=http"
  },
  "devDependencies": {
    "@google-cloud/functions-framework": "1.9.0"
  }
}

I think topselling_free collection is not available on Google Play anymore. I could see Top free, Top grossing, and Top paid on the website, but not top selling. I updated the tests to avoid this error.

dave-filion · 2022-06-22T14:41:27Z

@mmoksh thanks so much for this!

facundoolano · 2022-06-22T14:52:40Z

@mmoksh thanks for this.

This one PR addresses all the methods right? I'll set aside some time to review it, merge and publish a new version of the package. Please let me know if there something missing or broken still that I should take into account (I can't keep up with all the conversations going on in issues and PRs)

lib/app.js

lib/utils/appList.js

test/lib.app.js

test/lib.categories.js

mmoksh · 2022-06-23T01:10:50Z

@mmoksh thanks for this.

This one PR addresses all the methods right? I'll set aside some time to review it, merge and publish a new version of the package. Please let me know if there something missing or broken still that I should take into account (I can't keep up with all the conversations going on in issues and PRs)

Sure!
The PR covers all the methods.

lib/list.js

lib/app.js

facundoolano · 2022-06-23T14:35:39Z

lib/list.js

 const { BASE_URL } = require('./constants');
+const { processFullDetailApps } = require('./utils/processPages');
+
+function getBodyForRequests (collection, category) {


how is this function different from the one in utils/processPages ?

The body content and structure is different. We also pass different parameters here like the collection and category, not the token.

test/lib.app.js

mmoksh added 3 commits June 8, 2022 21:22

Fix search script

21dad6d

Fix app mapping - From PR#545

f05205d

Address comments on PR

ac2519a

mmoksh force-pushed the main branch from bcba150 to ac2519a Compare June 9, 2022 05:05

Fix mapping for script

fdf30c6

mmoksh force-pushed the main branch 2 times, most recently from 3a755cb to 76a54ff Compare June 10, 2022 05:43

Handle different sections in search results page

3dd5478

mmoksh force-pushed the main branch from 76a54ff to 3dd5478 Compare June 10, 2022 06:11

dbravobasecodeit mentioned this pull request Jun 12, 2022

Fixed Developer method, typo in extractDeveloperId #560

Closed

mmoksh mentioned this pull request Jun 13, 2022

Search is now consistently failing to return a result #553

Open

Fix gplay.list method

9ca47f7

Fix mappings for price field

dc2f92f

mmoksh changed the title ~~Fix mappings for search script~~ Fix the search, similar, and list methods Jun 18, 2022

mmoksh added 2 commits June 21, 2022 01:20

Fix tests

7f819f3

lint

5c609e3

Fix the method

8006736

mmoksh changed the title ~~Fix the search, similar, and list methods~~ Fix the search, similar, list, and developer methods Jun 22, 2022

facundoolano reviewed Jun 22, 2022

View reviewed changes

lib/app.js Outdated Show resolved Hide resolved

facundoolano reviewed Jun 22, 2022

View reviewed changes

lib/utils/appList.js Outdated Show resolved Hide resolved

facundoolano reviewed Jun 22, 2022

View reviewed changes

test/lib.app.js Outdated Show resolved Hide resolved

facundoolano reviewed Jun 22, 2022

View reviewed changes

test/lib.categories.js Outdated Show resolved Hide resolved

Address comments on code review

7efa03d

mmoksh requested a review from facundoolano June 23, 2022 01:12

nadavzipo reviewed Jun 23, 2022

View reviewed changes

lib/list.js Outdated Show resolved Hide resolved

facundoolano reviewed Jun 23, 2022

View reviewed changes

lib/app.js Outdated Show resolved Hide resolved

facundoolano added 2 commits June 23, 2022 11:30

Update lib/app.js

39fb700

Update lib/list.js

8feae8a

facundoolano reviewed Jun 23, 2022

View reviewed changes

test/lib.app.js Outdated Show resolved Hide resolved

Update test/lib.app.js

2aeebc4

facundoolano reviewed Jun 23, 2022

View reviewed changes

test/lib.app.js Outdated Show resolved Hide resolved

facundoolano added 2 commits June 23, 2022 11:49

Update test/lib.app.js

e51eed6

Update lib.app.js

e6cf76f

facundoolano merged commit e1e5439 into facundoolano:main Jun 23, 2022

This was referenced Jun 23, 2022

Fixed mapping for app listing #547

Closed

Add new mapping for google play changes #545

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix the search, similar, list, and developer methods #557

Fix the search, similar, list, and developer methods #557

mmoksh commented Jun 9, 2022 •

edited

Loading

Aliaksandr-Kasko-JazzTeam commented Jun 14, 2022

Aliaksandr-Kasko-JazzTeam commented Jun 14, 2022

NathanManning commented Jun 15, 2022

mmoksh commented Jun 16, 2022

Aliaksandr-Kasko-JazzTeam commented Jun 16, 2022

NathanManning commented Jun 17, 2022

mmoksh commented Jun 18, 2022

ninjachen commented Jun 20, 2022 •

edited

Loading

Aliaksandr-Kasko-JazzTeam commented Jun 20, 2022

mmoksh commented Jun 22, 2022

dave-filion commented Jun 22, 2022

facundoolano commented Jun 22, 2022

mmoksh commented Jun 23, 2022

facundoolano Jun 23, 2022

mmoksh Jun 23, 2022

Fix the search, similar, list, and developer methods #557

Fix the search, similar, list, and developer methods #557

Conversation

mmoksh commented Jun 9, 2022 • edited Loading

Aliaksandr-Kasko-JazzTeam commented Jun 14, 2022

Aliaksandr-Kasko-JazzTeam commented Jun 14, 2022

NathanManning commented Jun 15, 2022

mmoksh commented Jun 16, 2022

Aliaksandr-Kasko-JazzTeam commented Jun 16, 2022

NathanManning commented Jun 17, 2022

mmoksh commented Jun 18, 2022

ninjachen commented Jun 20, 2022 • edited Loading

Aliaksandr-Kasko-JazzTeam commented Jun 20, 2022

mmoksh commented Jun 22, 2022

dave-filion commented Jun 22, 2022

facundoolano commented Jun 22, 2022

mmoksh commented Jun 23, 2022

facundoolano Jun 23, 2022

Choose a reason for hiding this comment

mmoksh Jun 23, 2022

Choose a reason for hiding this comment

mmoksh commented Jun 9, 2022 •

edited

Loading

ninjachen commented Jun 20, 2022 •

edited

Loading